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Abstract 

The large-scale assessment of the English proficiency of prospective candidates is one 
of the integral aspects of the recruitment process of the Staff Selection Commission of the 
Govt, of India. The commission conducts the Combined Graduate Level Examination (CGL) 
in three stages or tiers to determine the selection of applicants to all Group B as well as non¬ 
technical Group C posts in the various ministries and departments of the government. Both 
Tier 1 and Tier 2 of the CGL examinations comprise sections that are intended to evaluate the 
English language skills of the applicants. This paper attempts to analyse select Tier 1 question 
papers of the CGL examination by employing Bachman and Palmer’s model of 
Communicative Language Ability. The model will be used to study the exam structure, 
evaluate the aspects of language ability that it prioritises, examine the validity of the item types 
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used in the test and the implications of these for the aspirants. In doing so, this paper also 
highlights the discrepancies in what the CGL exam purports to measure as outlined in the 
syllabus of the exam and what it measures in actuality. 

Introduction 

Taken by more than a million candidates every year, the Combined Graduate Level 
(CGL) Examination is a high-stakes exam conducted by the Staff Selection Commission of the 
Govt, of India. The exam is conducted in three stages or tiers and determines the selection of 
applicants to various non-gazetted posts in the government. This paper is an attempt to evaluate 
the English section of Tier 1 of the exam and attempts to study the syllabus of the exam to 
understand the language ability ‘construct’ that the exam purports to measure and proceeds to 
evaluate the question papers of the exam to analyse if the exam succeeds in measuring what it 
intends to measure. 

Methodology 

In order to evaluate the language ability ‘construct’ that the SSC exam purports to 
measure and compare it with the aspects of language ability that it evaluates in actuality, this 
paper employs Bachman and Palmer’s model of Communicative Language Ability (CLA) as 
outlined in their book A Handbook of English Language Testing which, according to the 
authors, can also be used as a language ability checklist to “judge the degrees to which 
components of language ability are involved in a given test or test task” (Bachman 77). 
According to Bachman and Palmer, language ability comprises of both language competence 
and strategic competence, each of which may be broken down into subcategories as shown 


below: 
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This paper concerns itself only with a few aspects of language competence namely 
grammatical knowledge, textual knowledge and socio-linguistic knowledge from the above 
model and compares how these abilities are evaluated across various question papers of the 
CGL exam. Each item in the sample sets of SSC question papers was mapped against the 
following checklist postulated by Bachman and Palmer (77) to identify the aspect of language 
ability in the checklist that is being tested by the item and items of the same type were compared 
across different instances of the exam the findings of which are reported in this paper. 


GRAM: Vocabulary 


GRAM: Syntax 


GRAM: Phonological/ Graphological 


TEXT: Cohesion 


TEXT: Rhetorical Organization 


FUNCT: Ideational 


FUNCT: Manipulative 
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FUNCT: Heuristic 
FUNCT: Imaginative 
SOCIO: Dialect 
SOCIO: Register 
SOCIO: Naturalness 

SOCIO: Cultural references and figurative 
language 

META: Goal Setting 
META: Assessment 
META: Planning 


Identifying the Construct from the Syllabus 

One of the first steps the researcher took in the analysis of the exam was to examine the 
syllabus of the exam as outlined by the Staff Selection Commission so as to be able to 
evaluate the language ability ‘construct’ for the proposed test. The syllabus of the English 
section of the CGL exam, stated in a single sentence in the Staff Selection Commission 
website, declares that the “candidates’ ability to understand correct English, his basic 
comprehension and writing ability, etc.” are the criteria that the exam intends to measure 
(“Combined Graduate Level Examination 2010”).The language ability construct for the exam 
has not been defined clearly and unambiguously which made it difficult for the researcher to 
evaluate the construct validity of the SSC exam. 

By correct English, is it implied that the candidate can identify right grammatical 
structures and spelling from incorrect ones? Does it also imply that candidates can pick the 
appropriate vocabulary for a given context? Given such a broad definition, one can only 
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make an educated guess that the exam intends to evaluate what according to Bachman and 
Palmer’s model constitutes grammatical knowledge and perhaps, socio-linguistic knowledge. 
Similarly, basic comprehension and writing ability are not operationalised in measurable terms. 
For instance, is the examinee expected to understand “the main points of clear standard input 
on familiar matters regularly encountered in work” (Common European Framework) according 
to the B1 level of the Common European Framework of Reference or are they expected to 
understand articles and reports on contemporary problems with particular viewpoints as 
expected of a candidate at the B2 level in the same framework? Similarly, what components of 
writing ability does the exam intent to evaluate? Does it evaluate the candidate’s knowledge 
of rhetorical organization or her sensitivity to various writing styles? The one-line syllabus of 
the Tier 1 exam fails to answer these questions. As the crucial question- “what does the exam 
intend to measure” could not be answered with clarity, the next logical step of this paper is to 
attempt an analysis of what the exam actually measures. 

Structure of the CGL exam 

The Tier 1 of the CGL exam is an objective exam of the multiple-choice type and 
comprises of a total of 100 questions on General Intelligence and Reasoning, General 
Awareness, Numerical Aptitude and English Comprehension. Though these sections are not 
demarcated clearly in the exam paper and one seamlessly merges into another, this paper 
looks at the section of the exam that evaluates the language ability of the participants which 
will be henceforth referred to as the ‘English section’ of the exam in this paper. 

The English section of the Tier 1 exam comprises 25 questions and the various item- 
types are given in a fixed sequence across all question papers of the exam. The following table 
is a representation of how nearly 86% of the sample question papers have been structured. As 
the ‘instructions’ column in the table below reveals, some item instructions that require that 
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the candidate identify the correctly spelled item from a list are phrased such that only one 
aspect of language-ability can be measured (spelling in this case) whereas other instructions 
such as the one that requires the candidate to “fill in the blanks” leaves scope for testing more 
than one language ability as syntax and vocabulary. The variance from the structure as well as 
the language ability being tested in a few instances will be discussed among the findings of this 
paper. 


Item Type 

No. of Questions 

Instructions 

Error Recognition 

2 

In the following question, 

some part of the sentence may 

have errors. Find out which 

part of the sentence has an 

error and select the appropriate 

option. If a sentence is free 

from error, select 'No Error'. 

Fill in the blanks 

2 

In the following question, the 

sentence given with blank to 

be filled in with an appropriate 

word. Select the correct 

alternative out of the four and 

indicate it by selecting the 

appropriate option. 

Synonyms 

2 

In the following question, out 

of the four alternatives, select 
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the word similar in meaning to 

the word given. 

Antonyms 

2 

In the following question, out 

of the four alternatives, select 

the word opposite in meaning 

to the word given. 

Idiom synonyms 

2 

In the following question, out 

of the four alternatives, select 

the alternative which best 

expresses the meaning of the 

idiom/phrase. 

Sentence Correction 

2 

In the following question, out 

of the four alternatives, select 

the alternative which will 

improve the bracketed part of 

the sentence. In case no 

improvement is needed, select 

"no improvement". 

Identify Synonyms 

2 

In the following question, out 

of the four alternatives, select 
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the alternative which is the 

best substitute of the phrase. 

Spelling 

2 

In the following question, four 

words are given out of which 

one word is incorrectly spelt. 

Select the incorrectly spelt 

word. 

Organize Sentences 

2 

The question below consists of 

a set of labelled sentences. Out 

of the four options given, 

select the most logical order of 

the sentences to form a 

coherent paragraph. 

Active Passive 

1 

In the following question, a 

sentence has been given in 

Active/Passive voice. Out of 

the four alternatives suggested, 

select the one which best 

expresses the same sentence in 

Passive/Active voice 

Direct/Indirect 

1 

In the following question, a 

sentence has been given in 

Direct/Indirect speech. Out of 
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the four alternatives suggested, 

select the one which best 

expresses the same sentence in 

Direct/Indirect speech. 

Cloze 

5 

In the following questions, the 

sentence given with blank to 

be filled in with an appropriate 

word. Select the correct 

alternative out of the four and 

indicate it by selecting the 

appropriate option. 


In an analysis of seven question papers from various sessions of the CGL conducted in 2017, 
the following patterns emerged. 

1. Same Item Types Evaluating Different Aspects of Language Ability 

One of the major findings of this analysis of question papers was that though the 
structure of the item-type remained the same across all iterations of the exam, the aspect of 
language ability evaluated by the same was not consistent across all question papers. 

1.1. Rhetorical Organisation or Syntax? 

For instance, the last 5 of the exam questions in 86% of the sample question papers 
are of the item type that is intended to test the candidate’s ability to logically organize 
sentences into paragraphs. These five questions have one common instruction phrased thus- 
“The question below consists of a set of labelled sentences. Out of the four options given, 
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select the most logical order of the sentences to form a coherent paragraph.” One such 
example is the CGL exam conducted on the morning of the 10th of August 2017 in which the 
above instruction was followed by the following question. 

P-But he did not know how to find one at that hour. 

Q-It was his first visit to the city and he didn't know where to go. 

R-Mohanlal's train was late and it reached Kolkata a little after midnight. 

S-He thought he would go to a choultry where he would not have to pay rent 

Options: 1) PSQR 2) QRSP 3) RQSP 4) RSQP (SSC CGL Tier 1) 

The above item requires that the candidate read all sentences, understand how they 
are connected to each other and arrange them in a logical order. It can be said that the above 
item tests the candidate’s reading comprehension and rhetorical organization. Most of the 
words, except “choultry” being common place, the language ability tested by the item is not 
vocabulary but his/her rhetorical organization at the textual level. 

However, the same item instructions were used incorrectly in about 14% of the 
question papers in the sample chosen by the researcher, one instance of which is the exam 
conducted on the 19 th of August 2017 in which the same instructions were used to create a 
different item as shown below: 

Then the gigantic figure 
X-alarming spectacle 
Y-slipped off the tree and stood 
Z-up before us, a truly 

Options: 1) YZX 2) YXZ 3) XZY 4) XYZ (SSC CGL Tier 1) 

Here, one can find that instead of testing the candidate’s ability for organising 
sentences to form a ‘text’ or a paragraph, this item tests the candidate’s control of the 
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syntactic structure at the sentence level. Here, what is being tested, according to Bachman 
and Palmer’s model, is the candidate’s organizational knowledge at the grammatical level, 
her syntax and not her ability to organize items at the textual level. Both items evaluate 
different ‘constructs’ as far as language ability is concerned-one at the textual level and one 
at the sentential level. 

1.2 Vocabulary Cloze or Grammar Cloze? 

The last five questions of the exam which comes to 20% of the English section of the 
exam is structured similar to the cloze test but unlike the cloze test in which words are 
“systematically removed ... and replaced with a blank” (McNamara 15), the item in the SSC 
exam had words randomly removed from a text. An analysis of this item across various 
instances of the SSC exam conducted in 2017 revealed this item primarily evaluated 
vocabulary in some instances of the SSC exam and grammar in others. 

For instance, in the afternoon session of SSC CGL exam held on the 17 th of August 2017 had 
the following question (Q196) of the cloze multiple-choice item type followed by five 
questions corresponding to each of the blanks in the passage: 

The Trikonasana, or triangle pose, stimulates the function of the entire body and 

_a lateral (side) stretch to the spine. It helps reduce blood pressure, 

stress, and anxiety. Practise this asana every day and you'll_strength in the 

ankles, thighs, knees, hips, calves and hamstrings. All standing poses_the 

cardiovascular system, so the more you do it, the_your stamina will 

become. The aim is never to overdo it, but to engage regularly,_that you 

get stronger and more stable over time. (SSC CGL Tier 1) 

The selected response options provided for the first blank of the above passage were 
“1) given 2) gave 3) to give 4) gives” (SSC CGL Tier 1). It can be observed that only one of 
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the options is grammatically correct which leads the researcher to the conclusion that the 
primary trait that is being assessed here is the candidate’s control over the structure/syntax of 
English and not his/her vocabulary. The same goes true for questions Q197, 198, 199 and 200 
of the exam. 

On the other hand, it can be observed that the afternoon session of the exam held on the 12th 
of August 2017 had the following question for the cloze item type: 

Management is a set of_that can keep a complicated system of people and 

technology running_. The most_aspects of management include planning, 

budgeting, organizing, staffing, controlling and problem solving. Leadership is a set 

of process that creates organization in the first place or adapts, them to_ 

changing circumstances. Leadership defines what the future should look like, aligns 

people with that_and inspires them to make it happen despite the obstacles. 

The selected response options provided to fill in the last blank of the above passage were the 
words “1) look 2) role 3) source 4) vision” (SSC CGL Tier 1). In contrast to the previous 
question, the selected response options were all grammatically correct. The candidate is 
therefore, tested for the ability to choose words that are best suited for the context from a 
given set of words. The primary trait that is being assessed here is the candidate’s vocabulary 
and not his/her control over the syntactic structure of the language. 

2. Different Item Types Across Examinations 

It has also come to the researcher’s attention that item types change across different 
instances of the exam. As mentioned earlier, 20% of the exam questions are of the cloze item 
type which usually constitute the last 5 questions in the English section of the exam. 


However, it can be observed that in some instances of the exam such as the one held on the 
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afternoon of the 23 rd of August 2017, the cloze item type is replaced by a reading 
comprehension question which consists of a passage followed by questions related to the 
passage. 

3. Evaluating Abilities Outside of Language 

The syllabus of the exam does not state that the topical knowledge of candidates will 
be evaluated. However, it can be observed that in several instances of the CGL exam, the 
abilities outside of language also are evaluated. 

3.1 Math 

Consider the following item which was part of the question paper of the CGL exam held on 
the afternoon of the 21 st of August 2017: 

Two centuries ago, the black rhinoceros - which roamed much of sub Saharan Africa 
- had 64 different genetic lineages; but today only 20 of these lineages remain, says 
the paper. The species is now restricted to five countries, South Africa, Namibia, 
Kenya, Zimbabwe and Tanzania. Genetically unique populations that once existed in 
Nigeria, Cameroon, Chad, Eritrea, Ethiopia, Somalia, Mozambique, Malawi and 
Angola have disappeared. The origins of the 'genetic erosion' coincided with colonial 
rule in Africa and the popularity of big game hunting. From the second half of the 
20th century, however, poaching for horns has dramatically depleted their population 
and genetic diversity, especially in Kenya and Tanzania. (SSC CGL Tier 1) 

This passage was followed by this question “Sub Sharan Africa has lost how many 
black rhino genetic lineages in 200 years (SSC CGL Tier 1)?” The question requires 
that the candidate have read the first sentence of the above passage and do a simple 
mathematical calculation to get the right answer. Though it be simple arithmetic that 


UELLH (International Journal of English Language, Literature in Humanities) Vol. 7, Issue 4, April 2019 


641 


the candidate is required to perform, such an item puts the construct being measured 
here under question and can be considered an “impure item”. J.B. Heaton, in 
providing guidelines for item writers creating multiple-choice questions warns against 
the use of ‘impure items’ and states that “It has long been standard practice to test 
only one item at a time: it is usually less confusing for a testee and it helps to 
reinforce a particular teaching point” (15), but we find the use of such items 
continuing in the CGL examination. 

3.2 Knowledge of the World 

The exam also tests constructs outside linguistic competence such as knowledge 
of the world as is illustrated in Question 73 which appeared in the morning shift of the 
SSC examination conducted on the 8 th of August 2017. The question contained the 

sentence “The higher the levels of ability and motivation the_the level of 

performance will be” which was followed by the options “higher, larger, lower, smaller” 
(SSC CGL Tier 1). In the above item, the distracters “larger” and “smaller” and are 
semantically inappropriate to use with the word “level” in this context. However, the 
adjectives, “higher” and “lower” can both be accurate in this context. That the candidate 
should pick “higher” is a test of his knowledge of the world that higher motivation 
usually leads to improved performance. The candidate’s knowledge of the world helps 
him easily eliminate options “3” and “4” which once again causes the researcher to 
question the language ability that is being measured here. Instead of the words Tower’ or 
‘smaller’, the words ‘bigger’, ‘mightier’ may have functioned as better distracters that 
would ensure that the ‘construct’ being assessed here is vocabulary and not only the 
candidate’s knowledge of the world. 


4. Vocabulary Items of Varying Difficulty Level Across Examinations 
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Fulcher and Davidson talk about archetypal ways of measuring particular skills and 
state that “an ‘archetype’ is a canonical item or task; it is the typical way to measure a 
particular target skill”(58). Though the SSC CGL exam uses the archetypal vocabulary 
question where candidates are required to find the synonyms of words, Though the term 
‘vocabulary’ does not find a mention in the syllabus of the CGL exam, items that test the 
vocabulary of candidates constitute a minimum of 6 out of 25 questions in the exam 
which comes to about 24% of the exam questions. This is without considering the items 
such as the cloze item and completion items which included vocabulary questions in 
many instances of the exam and syntax in a few instances as mentioned earlier. For 
instance, the exam conducted on the morning of the 8th of August had vocabulary items 
that required candidates to identify the synonyms of the words ‘frivolous’ and ‘petrify’ 
(SSC CGL Tier 1), which on the British National Corpus have a frequency of 213 and 12 
respectively. On the other hand, the exam conducted on the afternoon of the 12th of 
August required that candidates identify the synonyms of the words ‘boast’ and ‘haste’ 
(SSC CGL Tier 1), which have a frequency count of 1318 and 1316 respectively in the 
British National Corpus. This implies that the item may have been much easier for an 
examinee who took the test on the 12th as opposed to the 8th of August 2017. It was 
found in many instances of the cloze test item, the distracters were often not of the same 
level which implied that the cloze item may have tested language apart from vocabulary 
such as syntax and even the socio-linguistic competence of the candidate. 

5. Item Errors Compromising the Construct 

Apart from the above-mentioned findings where item constructs being evaluated 
differed across instances of the exam, it was also observed that even within a test, lack of 
adequate attention in creating items have occasionally compromised the construct being 
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measured. Consider the following cloze item (QID 98) from the exam conducted in the 
morning session of the 8th of August 2017 primarily intended to check the vocabulary of the 
candidate. 

Job performance is_by a number of factors. Motivation alone dos not lead to 

increased performance. Ability and technology moderates the relationship between 
motivation and performance. The higher the levels of ability and motivation the 

_the level of performance will be. However, increasing motivation beyond an 

_level tends to_a dysfunctional result because it is_by an 

increased level of anxiety. 

However, increasing motivation beyond an-level tends to produce a 

dysfunctional result because it is accompanied by an increased level of anxiety. 

1) certain 

2) desired 

3) increased 

4) optimal (SSC CGL Tier 1) 

In this case, the specifying of the article “an” before the blank compromises this item 
because it becomes easy for the candidate to eliminate the first two choices because they 
begin with a consonant sound. The testing of the aspect of language ability that requires that 
the candidate be able to use the appropriate indefinite article is not commensurate with the 
testing at the tertiary level as stated in the examination. The item thus tests phonology and the 
ability to choose between the articles “a” and “and” and does not pose adequate challenge as 
a vocabulary item to an examinee at the graduate level. 

In conclusion, it can be said that the language ability being evaluated by the SSC 
exam varies to a great extent in different forms or instances of the exam. It has been observed 
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that the ‘construct’ being evaluated by the exam is inadequately defined. Though the analysis 
of the exam makes it clear that the candidate’s grammatical knowledge and textual 
knowledge are being measured primarily and socio-linguistic knowledge to a small extent, 
these are measured inconsistently across various forms of the exam thus calling for this 
researcher to question the reliability of the exam. The researcher infers that item 
specifications for the exam have not been clearly drawn up thus causing different item writers 
to write items that evaluate different language abilities. The CGL being a high-stakes exam 
which determines the employment status of more than a million candidates every year and of 
which the government, the students and the larger population are potential stakeholders, it is 
in the interest of the nation that the exam states what it intends to measure in clear terms and 
succeeds in doing so to the maximum possible extent. In his work Practical Language 
Testing, Glenn Fulcher states that the “defence of high-stakes externally mandated tests is 
that they provide fairer access to opportunities and resources than any other method society 
has yet conceived” (8). On that note, this paper serves to function as a call to action to the 
govt, to re-examine its testing practices to ensure fairness to all the stakeholders of this 


examination. 
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